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Background 


DOD  Funding 


DOD  estimates  that  40%  of  the 
RDT&E  budget  is  spent  on  software 

(about  $30  billion  in  FY  06) 


Rest  of 
RDT&E 
budget 


Software 


budget 


As  much  as 
40%  of  software 
budget  is  spent 
on  rework 

(about  $12  B  in  FY  06) 
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Objectives, 
Scope,  and 
Methodology 


Objectives 

Identify  best  practices  and  metrics 
Analyze  causes  of  poor  outcomes 


Scope  and  Methodology 


POD  cases 

F/A  18  C/D  fighter  attack 
aircraft 

Tactical  Tomahawk  missile 

F/A-22  air  superiority  and 
ground  attack  aircraft 

Space-Based  Infrared 
System,  a  missile-detection 
satellite  system 

Comanche,  a  multi-mission 
helicopter 


Commercial 

cases 

Motorola 

NCR  Teradata 

General  Motors 

Computer 

Sciences  Corp. 
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Audit  Findings 


F-22  Experience 


Initially  put  together  a  sound  strategy  for 
software  development.  However . 

Significant  requirements  volatility 


RDT&E  estimate  grew  127% 
Cycle  time  grew  104% 
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Audit  Findings 


Comanche 

Experience 


u.s.  GAO 


Inadequate  Requirements  Analysis 
Requirements  Volatility 


231%  increase  in  RDT&E  cost  estimate 
120%  change  in  cycle  time 

CANCELLED 
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Uncontrolled  Requirements  Growth 
Optimistic  Reuse  Expectations 
Systems  Engineering  Deficiencies 


113%  Increase  in  RDT&E  Costs 
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Audit  Findings 

SBIRS 

Experience 


Audit  Findings 


Factors  Contributing  to 
Successful  Outcomes 


Useful 
metrics 


Disciplined 

processes 
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Manageable 

environment 


Audit  Findings 

Factors  Contributing 
to  Successful 
Outcomes 


General  Motors’ 

Development  Approach: 
Growing  Capability  Over  Time 


Manageable 

Environment 


Releases  with  new  functionality 

Releases  with  upgrades  to  existing  functionality 


Year  1 


Year  2 


Year  3 


Year  4 
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Audit  Findings 

Factors  Contributing 
to  Successful 
Outcomes 


Disciplined 

Processes 


Structured,  Gated  Reviews 


U.S.  GAO 


Gate  1 : 

Requirements 

review 


Gate  2: 
Design 
review 


Gate  3: 
Coding 
review 


Gate  4: 
Testing 
review 


Good  developers  don’t  just  wait  for  the  four  gated 
reviews  to  come  along;  they  proactively  review 
software  development  on  a  weekly  basis 
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Metrics 


Audit  Findings 


Factors  Contributing 
to  Successful 
Outcomes 
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Audit  Findings 

Factors  Contributing 
to  Successful 
Outcomes 

Useful 

_ „  Metrics 


F- 18  C/D 


Schedule 
|Requirements| 
Size 
Design 
Cost 
Quality 
Test 


F-22 


Schedule 


Design 


36%  RDT&E  Cost  Growth  127%  RDT&E  Cost  Growth 
6%  Schedule  Growth  104%  Schedule  Growth 
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Audit  Findings 

Factors  Contributing 
to  Successful 
Outcomes 

A  a.  Useful 
Metrics 


DOD  Program  Outcomes 
Linked  to  Management 
Controls 


Program 

Evolutionary 

environment 

Disciplined 

process 

Useful 

metrics 

%  change  in 
research, 
development, 
test,  and 
evaluation 
cost  estimate 

%  change 
in  cycle 
time 
estimate 

Tomahawk 

Yes 

Yes 

Yes 

7.6 

22.4 

F/A-18  C/D 

Yes 

Yes 

Yes 

36.4 

6.2 

F/A-22* 

No 

No 

No 

127 

104 

SBIRS* 

No 

No 

No 

113 

Not 

available 

Comanche* 

No 

No 

No 

231 

120 

*  GAO’s  assessment  addresses  conditions  found  before  these  programs  were  restructured 
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Conclusions 

Conclusions 

Software-intensive  weapon 
programs  are  more  likely  to  reach 
successful  outcomes  if  they  used 
evolutionary  environments  and 
disciplined  process  and  managed 
by  metrics.  Programs  that  did  not 
employ  these  practices 
consistently  garnered  poor  results 
from  software  acquisition. 
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Recommendations 


Recommendations 

Develop  a  list  of  systems 
engineering  deliverables 

Set  requirements  based  on 
systems  engineering 

Require  contractors  to  report  on 
seven  types  of  metrics 

Include  and  enforce  practices  in 
policies,  improvement  plans,  and 
development  contracts 
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DOD  Response 

DOD  Response 

Air  Force  adopted 
recommendations 

5000  Series  Acquisition  Policy  was 
amended  to  include  more 
emphasis  on  systems  engineering 
and  evolutionary  development 
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